{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Copyright 2022 Google LLC.\n",
        "\n",
        "SPDX-License-Identifier: Apache-2.0\n"
      ],
      "metadata": {
        "id": "kHHBibt259_6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# License"
      ],
      "metadata": {
        "id": "xgdUS5t-9On_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "\n",
        "\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "metadata": {
        "id": "73rCOT-_98ZE"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FUMRG2ESyyp4"
      },
      "source": [
        "# Voxel-based Radiance Fields in JAX and FLAX\n",
        "\n",
        "*Authors: Pedro Velez & Frank Dellaert*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJAe03h9z_Ob"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "Neural Radiance Fields (NeRF) exploded on the scene in 2020.\n",
        "\n",
        "A big new trend is the emergence of voxel-based, very fast NeRF variants, all foregoing the large MLP at the center of the original NeRF paper. They are listed here in order of increasing complexity:\n",
        "\n",
        "- [Plenoxels](https://alexyu.net/plenoxels) foregoes MLPs altogether and optimizes opacity and view-dependent color, using spherical harmonics (SH), directly on a 3D voxel grid. It stores 9 SH coefficients per vertex.\n",
        "\n",
        "- [ReLu Fields](https://geometry.cs.ucl.ac.uk/group_website/projects/2022/relu_fields/) is a very similar paper, but uses an additional ReLU on top of the density. The color uses SH as in Plenoxels.\n",
        "\n",
        "- [DVGO](https://sunset1995.github.io/dvgo/) is identical to Relu Fields for density, but replaces the color head with a small explicit-implixit hybrid MLP, feeding both position and view direction into the MLP, with positional encoding.\n",
        "\n",
        "- [Instant NGP](https://nvlabs.github.io/instant-ngp/) is the most complex architecture, and uses multiple voxelgrids and hashing as a novel way to deal with sparse densities.\n",
        "\n",
        "In this notebook we show how with jax/flax, it is relatively easy to quickly get a voxel-based NeRF variant up and running. Specifically, we will develop a simplified version of DVGO that directly regresses color instead of having a small MLP. It works remarkably well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gB5bj5jP4oOu"
      },
      "source": [
        "## Setup\n",
        "\n",
        "First, we install some of the necessary libraries. Of course, we need `jax` and `flax`, but we also install mediapy for visualization purposes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WUqU8lEO2aTl",
        "outputId": "1accafb5-0f39-4715-aad3-b8bbad1e0d4e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "# Install the latest JAXlib version.\n",
        "%pip install --upgrade -q pip jax jaxlib \n",
        "# Install Flax at head:\n",
        "%pip install --upgrade -q git+https://github.com/google/flax.git\n",
        "%pip install --upgrade -q git+https://github.com/google-research/sunds.git\n",
        "%pip install -q mediapy"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we import all of these, and then some."
      ],
      "metadata": {
        "id": "udtnFDYz4k8f"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "ooWdY4VB1md8"
      },
      "outputs": [],
      "source": [
        "from typing import Callable, Tuple\n",
        "Initializer = Callable\n",
        "\n",
        "from functools import partial\n",
        "import dataclasses\n",
        "import time\n",
        "\n",
        "import tqdm\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bmfwEB631lT2",
        "outputId": "79fa3016-4361-46ca-a16e-f3c2468228e2"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[StreamExecutorGpuDevice(id=0, process_index=0, slice_index=0)]"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "\n",
        "import jax\n",
        "import jax.numpy as jnp\n",
        "from jax import random\n",
        "jax.devices()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "O2rNz6G9zaOJ"
      },
      "outputs": [],
      "source": [
        "import flax\n",
        "import flax.linen as nn\n",
        "from flax.core import freeze, unfreeze\n",
        "from flax.core.frozen_dict import FrozenDict\n",
        "\n",
        "import optax"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "P_bBItMkjRlv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8d4cf5c1-def6-4414-cc24-f1d813841664"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "mkdir: cannot create directory ‘/root/tensorflow_datasets/’: File exists\n"
          ]
        }
      ],
      "source": [
        "import sunds\n",
        "import mediapy as media\n",
        "import tensorflow_datasets as tfds\n",
        "# load synthetic nerf dataset locally\n",
        "!mkdir ~/tensorflow_datasets/\n",
        "!gsutil -q -m cp -r gs://kubric-public/tfds/nerf_synthetic_*/ ~/tensorflow_datasets/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TlxfMEtHQCWF"
      },
      "source": [
        "## Differentiable interpolation in 1D\n",
        "\n",
        "We start simple, in 1D, by building a small differentiable 1D grid. The class `LineGrid` below is a minimal flax.linen module with variables, that are linearly interpolated at an input location. If the grid contains multi-dimensional feature vectors, the output has the same dimension: features are interpolated independently of one another.\n",
        "\n",
        "The grid is always defined at points with coordinates 0.0 to float(size-1), e.g., LineGrid(size=5, features=3) will define 3D features at the locations 0.0, 1.0, 2.0, 3.0, and 4.0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "-RTUmimX8ds4"
      },
      "outputs": [],
      "source": [
        "class LineGrid(nn.Module):\n",
        "  size: int\n",
        "  features: int = 1\n",
        "  grid_init: Initializer = nn.initializers.lecun_normal()\n",
        "\n",
        "  @staticmethod\n",
        "  def interpolate(feature_grid, x):\n",
        "    \"\"\"Interpolate vectors on a regular grid at samples 0, 1, ... n-1.\"\"\"\n",
        "    xd, whole = jnp.modf(x) # 3.6 -> xd = 0.6\n",
        "    x0 = whole.astype(int)  # x0 = 3\n",
        "    x1 = x0 + 1             # z1 = 4\n",
        "\n",
        "    def f(grid):\n",
        "      return grid[x0] * (1.0 - xd) + grid[x1] * xd\n",
        "\n",
        "    return jax.vmap(f, -1, -1)(feature_grid)\n",
        "\n",
        "  @nn.compact\n",
        "  def __call__(self, x):\n",
        "    grid = self.param('grid', self.grid_init, (self.size + 1, self.features))\n",
        "    return self.interpolate(grid, x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sl-2a8wIAH3j"
      },
      "source": [
        "## Training a LineGrid\n",
        "\n",
        "Training in JAX is standard and easy: we set up a mean-squared error loss (MSE) and a train function that runs a specified number of epochs. Below we use standard stochastic gradient descent (SGD) from the `optax` library."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "yfq8F9Ki3n5N"
      },
      "outputs": [],
      "source": [
        "# Same as JAX version but using model.apply().\n",
        "@jax.jit\n",
        "def mse(variables, x_batched, y_batched):\n",
        "  # Define the squared loss for a single pair (x,y)\n",
        "  def squared_error(x, y):\n",
        "    pred = model.apply(variables, x)\n",
        "    return jnp.inner(y-pred, y-pred) / 2.0\n",
        "  # Vectorize the previous to compute the average of the loss on all samples.\n",
        "  return jnp.mean(jax.vmap(squared_error)(x_batched, y_batched))\n",
        "\n",
        "def train(variables, x_samples, y_samples, learning_rate, num_epochs, checkpoint_freq=100):\n",
        "  tx = optax.sgd(learning_rate=learning_rate)\n",
        "  opt_state = tx.init(variables)\n",
        "  loss_grad_fn = jax.value_and_grad(mse)\n",
        "\n",
        "  for i in range(num_epochs):\n",
        "    loss_val, grads = loss_grad_fn(variables, x_samples, y_samples)\n",
        "    updates, opt_state = tx.update(grads, opt_state)\n",
        "    variables = optax.apply_updates(variables, updates)\n",
        "    if i % checkpoint_freq == 0:\n",
        "      print('Loss step {}: '.format(i), loss_val)\n",
        "  \n",
        "  return variables"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we first create a \"ground truth\" 1D grid, stored in `true_variables`, and then generate some training samples from it, directly using the `interpolate` method that we defined above. Because the LineGrid is fully differentiable, we can then regress the `trained_variables` from those samples, using a single call to `train`."
      ],
      "metadata": {
        "id": "ilSXeRqC5ZrQ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5dCwzhl3-vpr",
        "outputId": "209d678c-dda9-4b1e-d7d7-5fd768a0e2eb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss step 0:  63.487186\n",
            "Loss step 100:  1.5596838\n",
            "Loss step 200:  0.6766525\n",
            "Loss step 300:  0.52439857\n",
            "Loss step 400:  0.4857689\n",
            "Loss step 500:  0.47576818\n",
            "Loss step 600:  0.4731633\n"
          ]
        }
      ],
      "source": [
        "# Generate expected grid and ground truth\n",
        "grid_size = 10\n",
        "grid_values_scale = 2\n",
        "true_grid = np.arange(grid_size+1).reshape((grid_size+1,1)) * grid_values_scale\n",
        "true_variables = freeze({'params': {'grid': true_grid}})\n",
        "\n",
        "# Generate training data\n",
        "num_x = 128\n",
        "key1, key2, key3, key4 = random.split(random.PRNGKey(42), num=4)\n",
        "x_samples = random.uniform(key1, (num_x, 1)) \n",
        "x_samples += random.randint(key2, (num_x, 1), minval=0, maxval=grid_size)\n",
        "y_samples = LineGrid.interpolate(true_grid, x_samples) + random.normal(key3, (num_x,1,1))\n",
        "\n",
        "# Setup model and initialize parameters\n",
        "model = LineGrid(size=grid_size)\n",
        "variables = model.init(key4, x_samples)\n",
        "\n",
        "trained_variables = train(variables, x_samples, y_samples, learning_rate=0.3, num_epochs=601)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we show the result in red, along with the training samples in blue. This is the case where the grid stires scalar values, but we can just as well regress using vector-based values. This will come in handy when we later regress to RGB colors in the NeRF case."
      ],
      "metadata": {
        "id": "Dj-RNVeB5z8C"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 537
        },
        "id": "KfEdWkbL-PWF",
        "outputId": "c93bbf35-2515-4ef7-9276-50f58bdcccf8"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x648 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr8AAAIICAYAAABn1oYjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdf3yddX338fc3P6DnQGug7YAEk/TeXBRa2khwsiIgGwuryGLH5u2OUHGYQXCC00BqnKISWyxT62qLgSGlXFMmdpFJR50Wh3TOB+mdctfiunmPJJIySQtZcUkhP677jysnOefkXOdc59d1fr2ej8cep+d7rpxzkQx895vP9/Mxtm0LAAAAKAcV+b4BAAAAwC+EXwAAAJQNwi8AAADKBuEXAAAAZYPwCwAAgLJB+AUAAEDZqPLzw5YtW2Y3Njb6+ZEAAAAoQwcOHDhm2/by2HVfw29jY6P6+/v9/EgAAACUIWPMULx1yh4AAABQNgi/AAAAKBuEXwAAAJQNX2t+45mcnNQLL7ygkydP5vtW4LNFixbp3HPPVXV1db5vBQAAlIm8h98XXnhBixcvVmNjo4wx+b4d+MS2bR0/flwvvPCCVqxYke/bAQAAZSLvZQ8nT57U0qVLCb5lxhijpUuXsuMPAAB8lffwK4ngW6b4uQMAAL8VRPjNt9NPP12SdPToUV177bVpv8/U1JQ+8YlP6E1vepPWrFmjNWvWqKenx/X6devWaWxsbMH6nXfeqXvuuSfuel1dndasWaOVK1fqscceS/teBwcHtXLlSklSf3+/PvKRjyS8/vOf/3zU89/+7d9O+7MBAADyhfAboba2Vo8++mjaX//JT35SR48e1aFDh3Tw4EH96Ec/0uTk5ILrbNvWzMyM9uzZo5qampQ+46Mf/agOHjyob33rW/rgBz+omZmZqNenpqZSvu+WlhZ95StfSXhNbPj9l3/5l5Q/BwAAIN+KL/xaltTYKFVUOI+WlbW3jtwNffDBB7V+/XpdddVVetOb3qTbb7997rrvfe97uvjii/XWt75Vf/RHf6Rf/epXGh8f13333ae//uu/1qJFiyRJixcv1p133jn33k1NTbr++uu1cuVK/eIXv1BjY6OOHTsmSerp6dFv/uZv6pJLLtGRI0eS3utb3vIWVVVV6dixY7r88st12223qaWlRVu3btWBAwd02WWX6cILL1Rra6tefPFFSdKBAwe0evVqrV69Wl/96lfn3uuHP/yhrr76aknSr371K91www1atWqVLrjgAn37299WV1eXJiYmtGbNGoVCIUnzu+W2bauzs1MrV67UqlWr9Mgjj8y95+WXX65rr71Wb37zmxUKhWTbdto/GwAAgGzIe7eHlFiW1N4ujY87z4eGnOeSNBvKsungwYMaGBjQqaeeqqamJv35n/+5AoGA7rrrLn3/+9/Xaaedprvvvltf/OIX1dbWpvr6ei1evNj1/f7jP/5DO3fu1Nvf/vao9QMHDuib3/ymDh48qKmpKb31rW/VhRdemPDefvKTn6iiokLLlzsjq19//XX19/drcnJSl112mb7zne9o+fLleuSRR9Td3a0HHnhAN9xwg7Zt26ZLL71UnZ2dcd/3c5/7nN7whjfo0KFDkqRXXnlFf/iHf6ht27bp4MGDC67fvXu3Dh48qGeffVbHjh3TRRddpEsvvVSSNDAwoMOHD6u2tlZr167V/v37dckllyT85wIAAMil4gq/3d3zwTdsfNxZz0H4/Z3f+R294Q1vkCSdd955Ghoa0tjYmJ577jmtXbtWkhM6L7744gVf+/Wvf11bt27V8ePH50oEGhoaFgRfSfrRj36k97znPQoGg5Kka665xvWevvSlL+nhhx/W4sWL9cgjj8wdGnvve98rSTpy5Ih++tOf6sorr5QkTU9P65xzztHY2JjGxsbmgul1112nf/zHf1zw/t///vf1zW9+c+75GWeckfB79PTTT+t973ufKisrddZZZ+myyy7TM888oyVLluhtb3ubzj33XEnSmjVrNDg4SPgFAAB5VVzhd3g4tfUMnXrqqXN/rqys1NTUlGzb1pVXXqlvfOMbUdeOj49reHhYr776qhYvXqwbbrhBN9xwg1auXKnp6WlJ0mmnnZbxPX30ox/Vxz/+8QXr4fe2bVvnn3++fvzjH0e9Hu9gXa7F+/4BAADkU3HV/NbXp7aeA29/+9u1f/9+/fznP5ck/c///I/+/d//XcFgUH/6p3+qD3/4w3O9a6enp/X6668nfc9LL71UfX19mpiY0Kuvvqp/+Id/SPv+mpqaNDo6Ohd+JycndfjwYdXU1KimpkZPP/20JMlyqZW+8soro+qBX3nlFUlSdXV13MN773jHO/TII49oenpao6Ojeuqpp/S2t70t7fsHAADIpeIKvz090mxpwJxg0Fn3yfLly/Xggw/qfe97ny644AJdfPHF+rd/+7fZ2+vROeeco5UrV6q5uVnveMc7tGHDBtXW1iZ8z7e+9a1673vfq9WrV+v3f//3ddFFF6V9f6eccooeffRR3XHHHVq9erXWrFkzV3bx9a9/XbfccovWrFnjevjsk5/8pF555RWtXLlSq1ev1pNPPilJam9v1wUXXDB34C3sPe95jy644AKtXr1aV1xxhb7whS/o7LPPTvv+AQAAcsn4eQK/paXF7u/vj1r72c9+pre85S3e38SynBrf4WFnx7enJyf1vvBHyj9/AAAAD4wxB2zbboldL66aX8kJuoRdAAAApKH4wi8AAAAKWt/AiLbsPaKjYxOqrQmos7VJbc11+b4tSYRfAAAAZFHfwIg27j6kiUmn29XI2IQ27nbmBxRCAC6IA29M/ipP/NwBACg9W/YemQu+YROT09qyN/kEWz/kPfwuWrRIx48fJwiVGdu2dfz48blR0AAAoDQcHZtIad1veS97OPfcc/XCCy9odHQ037cCny1atGhuAhwAACgNtTUBjcQJurU1gTzczUJ5D7/V1dVasWJFvm8DAAAAWdDZ2hRV8ytJgepKdbY25fGu5uU9/AIAAKB0hA+10e0BAAAAZaGtua5gwm6svB94AwAAAPxC+AUAAEDZIPwCAACgbBB+AQAAUDYIvwAAACgbhF8AAACUDcIvAAAAygbhFwAAAGWDIRcAAAAlpm9gpGAnrOUb4RcAAKCE9A2MaOPuQ5qYnJYkjYxNaOPuQ5JEABZlDwAAACVly94jc8E3bGJyWlv2HsnTHRUWwi8AAEAJOTo2kdJ6uSH8AgAAlJDamkBK6+WG8AsAAFBCOlubFKiujFoLVFeqs7UpT3dUWJKGX2PMG40xTxpjnjPGHDbG3Dq7fqYx5p+MMf8x+3hG7m8XAAAAibQ112nT+lWqqwnISKqrCWjT+lUcdptlbNtOfIEx50g6x7bt/2OMWSzpgKQ2SR+Q9LJt25uNMV2SzrBt+45E79XS0mL39/dn584BAAAAF8aYA7Ztt8SuJ935tW37Rdu2/8/sn1+V9DNJdZL+QNLO2ct2ygnEAAAAQMFKqebXGNMoqVnSTySdZdv2i7Mv/Zeks1y+pt0Y02+M6R8dHc3gVgEAAIDMeA6/xpjTJX1b0m22bZ+IfM12aifi1k/Ytt1r23aLbdsty5cvz+hmAQAAgEx4Cr/GmGo5wdeybXv37PIvZ+uBw3XBL+XmFgEAAIDs8NLtwUj6G0k/s237ixEvPSZpw+yfN0j6TvZvDwAAANnQNzCitZv3aUXX41q7eZ/6BkZy92EdHVJVlWSM89jRkbvPSlGVh2vWSrpO0iFjzMHZtU9I2izp74wxfyppSNIf5+YWAQAAkIm+gRFt3H1obuzxyNiENu4+JEnZb4HW0SHt2DH/fHp6/vn27dn9rDQkbXWWTbQ6AwAA8N/azfs0Eme8cV1NQPu7rsjuh1VVOYE3VmWlNDWV3c9KIO1WZwAAAChuR+MEX8nZAc56+UO84Jto3WeEXwAAgBJXWxNwfW3j7kPZDcCVlamt+4zwCwAAUOI6W5sUqI4fPicmp7Vl75HsfVh7e2rrPvNy4A0AAABFLHyo7bZHDsZ93a0sIi3hQ229vU6pQ2WlE3wL4LCbxM4vAABAWWhrrlOdS/lDorKItGzf7hxus23nsUCCr0T4BQAAKBvxyh8C1ZXqbG3K0x35j7IHAACAMhEuf9iy94ha9u/Rxqd36az/HpXZVS/19EihUJ7vMPcIvwAAAGWk7bkfqu2eW2UfPy4TXhwa0tSNH3KCYYkHYMoeAAAAcsjXscKJWJa0bJn0/vdLkcF3VtXJCY133pGXW/MTO78AAAA54mWscN/AiLbsPaKjYxOqrQmos7Up85HDliV1d0vDw1J9vbRunbRzpzQ+nvDLFr14NLPPLQLs/AIAAOTIlr1H5oJvWGRf3XA4HhmbkK35cJzR7nBHh3TdddLQkNNtYWhIuvfepMFXko4uWZb+5xYJwi8AAECOuPXPDa8nC8cpsywn6Np29Hrs8zjGq07V/VfdmN7nFhHCLwAAQI649c8NrycLxynr7vYUdCPZko4vWqxPXf0Rrem6Jb3PLSKEXwAAgBxJ1lc3WThO2fCw+2sm+oibLWksuES3Xf0xXfPpPl3yqVszrzUuAoRfAACAHGlrrtOm9atUVxOQkVRXE9Cm9avmQmbKQycsS2pslCoqnEfLin69vj7+1xkj3XST1NDg/LmhQebhh1XzP/+trf9wj/Z3XVEWwVei2wMAAEBOtTXXuQbLyKETSbs9WJbU3j5/cG1oyHkuzffm7emJvkaaD74FNGI4n4ydYl1IJlpaWuz+/n7fPg8AAKBkNDY6gTdWQ4M0ODj39JmebXrjPZ/Tr42N6qWa5frFx/9SF3V/2LfbLBTGmAO2bbfErrPzCwAAkEUZ9e2N7c8bOXLYrZ43Yr1vYEQbT/6GJv7sgbm1wMlKbRoY8bWsISe9i7OE8AsAAJCGeAFPUtKhFguEA+/QkFOiEP6tfGxZQ319/J3fiDrfRK3T/AqfXgZ75BMH3gAAAFLkNpzizscOp9a3N1zHGw61seWo4+NOMJacXeBgMPr1YNBZn5X11mlpyHrv4iwj/AIAAKTILeCNTUzGvd41fHZ3J5+8Fi5rCIWk3t6ojg3q7Z0vi1AOWqeloRACeCKEXwAAgBSlEuSuOfykfvy1D8ZvT5aoL29YZPuyUMg53DYz4zxGBF8pjdZpOVAIATwRwi8AAECK3ILcGcHqqPB5zeEndffebTp77CWnpCFcxxsOwG59ecNiyhqSSdZX2A+FEMATodUZAABAimIPdUlOwNu0fpWk+b69P/7aB53gGyvcniy2d680f+itoSG620MRKYRuD26tzgi/AAAAaXANeJHtytxyljFO6YKUuL0Z0kb4BQAAyKG+gREd3PxV3b77iwpOvZb44pjBFMg+hlwAAICyl6tfx/cNjOjpz27V5u/8larsmcQXp1jHi+ziwBsAACgLbr15+wZGMn7vg5u/qs9+9yuJg69LezL4i51fAABQFnIy/Wy2XvfTQ0Myia6jzKFgEH4BAEBZyPrwhY4O6d57JdtOHHwpcygohF8AAFAWamsCGokTdBMNX0jY0WE2+CYyU1GpCsocCgo1vwAAoCykOnwhYY1wd3fS4Du1KKCKh3YSfAsM4RcAAJSFVKefJaoRTjqWuKFBVfffR/AtQJQ9AACAstHWXOf5cFu4Fviaw0/q9qceUu2JYzq6ZJm2XHq9M4xiaGjhFxkj7dpF6C1g7PwCAADEUVsT0Gf2bteXv/tXOvfEqCpk69wTo9q8d5u0bp1zkC2SMdJNNxF8CxzhFwAAII4vTz+n6w7uWRCWApOvSXv2OP16Gxrm+/fu2iVt356Xe4V3jDcGAACIp7ExfmmD5ATemSST3JBXbuON2fkFAACIZFmJg6+k8bNr/bsfZBXhFwAAIMyypPb2hMF3RtIX3nG9f/eErCL8AgAAhHV3S+Pjri/PSNq1Zp12rljr3z0hq2h1BgAAEObSv9eWNLJkub5w6fV67Px3qi7BVDgUNsIvAABAmEv/3pEly3XJzV+XlHgqXKZcxyl7fB3JUfYAAAAQ1tOzoH/v1KKA7r/qRk9T4TKRcJyyh9fhDeEXAACUj3Anh4oK59Gyol8PhRb07626/z7d+cjn9fzmd2l/1xU522lNOE7Zw+vwhrIHAABQVFL91X/4+pb9e7R57zZnSIXklDe0tzt/jpzKFgrlZUpbeJyy23qy1+ENO78AAKBopPqr/8jrO596aD74ho2POx0eCkCtyyG68Hqy1+EN4RcAABSNVH/1v2XvEV158Pt6escNqjsxGv9NXTo8+K2ztUmB6sqotcjDdclehzeUPQAAgKKR6q/+W/bv0aYntik49Vrc1yU5HR4KQLh0w62kI9nr8IbwCwAAikZtTUAjcYKu26/+Nz69K3HwDQadDg8Foq25LmGYTfY6kqPsAQAAFI1Uf/V/1n/HL3WwJaejQ29vXg63IX/Y+QUAAEUj1V/9G5ehFaahQRoczPh+GDpRfIxt2759WEtLi93f3+/b5wEAgDJiWU7nhuFhp443XM7Q3u50dQgLBhPu+HoNtOFOEpEH8ALVlTkbgoHUGGMO2LbdErtO2QMAACh+luWE3KEhybaje/jGDK1IFny9tlJj6ERxouwBAAAUv+7u6N1dab6H7+Cg57reRIE2djeXoRPFiZ1fAABQ/Nx69abYwzeVQMvQieJE+AUAAMXFsqTGRqmiwnm0LPdevSn28E0l0DJ0ojgRfgEAQFr6Bka0dvM+reh6XGs373MdMZxVbrW969Y5B9kipdHDN5VA29Zcp03rV6muJiAjqa4mwGG3IkC3BwAAkLK8dTpobIzbukwNDU7Qje32kEYPX9qXlQa3bg+EXwAAkLK1m/fFnbRWVxPQ/q4rcvfBFRXOjm8sY6SZmdx9LooOrc4AAEDW5K3TQZZqe1G+CL8AAORRXupmsyBvnQ56erJS24vyRfgFACBPUhmoUGjy1ukgFEppaAUQiyEXAADkSSoDFQpN+P7ycjAsFCLsIm2EXwAA8qTYJ4S1NdcVfEgHYlH2AABAnjAhDPAf4RcAgDxhQhjgP8oeAADIk7zWzQJlivALAEAeUTcL+IuyBwAA4C/LcsYUV1Q4j5aV7ztCGWHnFwAA+MeypPZ2aXzceT405DyXMm5f1jcwQgkJkjJ2vPnYOdLS0mL39/f79nkAACA1OQ+QjY1O4I3V0CANDqb9tuGBIZF9kwPVldq0fhUBuEwZYw7Ytt0Su07ZAwAAkOTTxLnh4dTWPUo0MASIRPgFAACSfAqQ9fWprXtU7AND4B/CLwAAkORTgOzpkYLB6LVg0FnPAAND4BXhFwAASPIpQIZCUm+vU+NrjPPY25vxYTcGhsArwi8AAJDkY4AMhZzDbTMzzmOGwVdy+iVvWr9KdTUBGUl1NQEOuyEuWp0BAABJxT9xjoEh8ILwCwAA5qQUIC1L6u52OjXU1zt1u1nYxQVyifALAABSl8GwCoZRIJ+o+QUAAKnr7p4PvmHj4856Ar70EgYSIPwCAIDUpTmsgmEUyDfCLwAASF2awyoYRoF8I/wCAIDUpTmsgmEUyDfCLwAASF2awyoYRoF8o9sDAAAlLmfdFUKhlFubFXsvYRQ/wi8AACUs3F0hfMgs3F1BUvzA6UPvXoZRIJ8oewAAoIR57q7Q0SFVVkrvf7/Ts9e253v3WpaPdwzkFuEXAIAS5qm7QkeHtGOHNDOz8EIPvXuBYkLZAwAARcpLLW9tTUAjcQJwVHeF3t7EH5Skdy9QTNj5BQCgCHmdlOapu8J0dFnEAkl69wLFJGn4NcY8YIx5yRjz04i1O40xI8aYg7P/ty63twkAACJ5reVta67TpvWrVFcTkJFUVxPQpvWroneIK6PDcRQPvXuBYuKl7OFBSdskPRSz/iXbtu/J+h0BAICkUpmUlrS7Qnu7U/Mb67TTpK99LevdHoB8Srrza9v2U5Je9uFeAACARylNSrMsqbFRqqhwHmO7N2zfLt188/wOcGWl8/xXvyL4ouRkUvP7YWPM/50tizgja3cEAACS8jwpraNDuu665O3Ltm+Xpqaca6amnOdACUo3/O6Q9OuS1kh6UdJfuV1ojGk3xvQbY/pHR0fT/DgAABDJUy2vZUn33usE2ki0L0MZM3bsvxDxLjKmUdJ3bdtemcprsVpaWuz+/v6UbxIAAKShsdHZ6Y3HmPh9fYESYYw5YNt2S+x6Wju/xphzIp6+R9JP3a4FAAB5kqg/L+3LUKaSdnswxnxD0uWSlhljXpD0aUmXG2PWSLIlDUr6sxzeIwAASEd9ffydX2NoX4aylTT82rb9vjjLf5ODewEAYAEvU8zgoqfHOdw2Pj6/Zox00010cUDZYsIbAKBgeZ1iVraStTALhZzRxQ0NTuhtaJB27YrbyaFvYERrN+/Tiq7HtXbzPr7HKFlehlwAAJAXiaaYlfXur2VJt94qHT8+vxZuYSZF7er2nXe5ttz0wPzO+XlNaot5u76BEXV+61lNzjiH4EfGJtT5rWclqby/zyhJ7PwCAApWKlPMyoZlaerGD0UH37CYFmZed87vfOzwXPANm5yxdedjh3PxTwDkFeEXAFCwUppiVibGO+9Q1ckE4T+iw0OinfNIYxOTcd/KbR0oZoRfAEDB8jzFrIwsevFo4gsiWpixcw4sRPgFABQsT1PMyszRJcvcXwwGo1qYed05PyNYHfc6t3WgmBF+AQAFra25Tvu7rtDzm9+l/V1XlHXwlaT7r7pR41WnRq3ZksaCS5zODhGH3bzunH/63eerutJErVVXGn363edn9+aBAkD4BQCgiKzpukWfuvojemHJcs3I6IUly9XZ1qkfPv3cgt69XnfO25rrtOXa1VHXbbl2ddn/RQOlydi2nfyqLGlpabH7+/t9+zwAAEoRgz+A5IwxB2zbboldp88vAABFpq25jrALpImyBwAAcijp5LRkU9oAZBU7vwAA5Eh4yES41254yIQ0Ozmto0O6914pXILoMqUNQPaw8wsAQI4kHDJhWbJ3RATfsJgpbQCyi51fAAByJHaYxDWHn9TtTz2k2hPHNFNRoQq5HDqPmNIGILvY+QUAIEcih0lcc/hJbX5im849MaoK2aqYmXb/wogpbQCyi51fAABypLO1SU9/dqtu2/eg6k6MyiT/Es1IqoiY0gYguwi/AADkSNtzP9TVT2xT1cmJ5BfLCb5//1vX6A857AbkDGUPAADkSnd30uA7ZSrmJrXd3tapyh3bfbo5oDyx8wsAQK4kObg2tSigu665TTtXrGVSG+ATwi8AANlgWU6LsuFh58BaT4/zODQU//qGBlX19OjOUEh3+nqjQHmj7AEAgHSFp7MZI113nRN0bXt+WMW6dVIwGP01waD08MPS4CCDLIA8IPwCAJAOy3ICbnhnN96wij17pN5eqaHBCcgNDc5zQi+QN8aO/Zc1h1paWuz+/n7fPg8AgJxpbHQvaQgzRpqZ8eV2AEQzxhywbbsldp2dXwAA0uFlChvDKoCCw4E3AEDJ6hsY0Za9R3R0bCL73RQSHWaTnNpehlUABYedXwBASeobGNHG3Yc0MjYhW9LI2IQ27j6kvoERb28QPsxWUeE8Wlb06z09Cw+zmdkZbtT2AgWL8AsAKElb9h7RxOR01NrE5LS27D2S/IsjD7NFdm+IDMCh0MLDbLt2OdfTyQEoWIRfAEBJOjoWf7Ka23rUTu+GDU63hkjj404f30ihkBN0Z2YIvECRoOYXAFCSamsCGokTdGtrAgsvtixN3fih+VHE09MLr5FkDw/LuHxeTuuLAWQNO78AgKLRNzCitZv3aUXX41q7eV/C+t3O1iYFqiuj1gLVlepsbVpw7XjnHfPBN4FfvmG5631lVF8MwDeEXwBAUUg1YLY112nT+lWqqwnISKqrCWjT+lXzu7ERZQ6BF5OH1PGqU7XpkuvivpZRfTEAX1H2AAAoCokCplt5QVtzXfzXwgfaZut63UoZpkyFKmxbR5cs0xcuvV4H1q6Le13K9cUA8obwCwAoChkHTMtyDqwNDzuH2lzqesPGq05V11Uf1mPnv1OSUzKxKU7JhJRifTGAvKLsAQBQFNyCpKeAGdu6zO1Am6QZGb2wZLm61/25nmr5vfglEzFSqS8GkF/s/AIAikJna5M27j4UVfqQNGB2dDi9eJPs8ob9subXdPGfPTDXreFLHrs1hEMx3R6Awkf4BQAUhZQDZkeHtGOH9w8IBnX2ti/q+dC70r6/ZGGXdmhA/hF+AQAFxy0kJgyYkTW99fXOYzKVlc6Aivp6Z1xxDodUhLtVhHeuw90qJBGAAR9R8wsAKChp9cyNN47YthN/UDAo7dzp23Q22qEBhYHwCwAoKGmFxO7uheOIE2lo0DOfuFtrf3GOp4EZ2UA7NKAwEH4BAHNSmaCWK2mFRC8lDmE336y+v9+v60/+hq8T2TLqVgEgawi/AABJhTOiN62QWF8ff/300526Xsl5vPlmafv2vJQg0A4NKAyEXwCApMKpSU0YEiNGEqux0XkuOYfVgsHoNwoG9UzXJq2963taccd3tfau76nvQ92S8lOCkHTcMgBf0O0BACCpcGpSY1uabXh+v27/0UMK3jUiGTN/kG1oyDnkJs0fVovo9vDMhz6u60/+hiYmnfuP7K6QrYlsqbYu89IODUBusfMLAJBUWDWpbc112t91hZ5fNaY7v7tVwRdnSy9iOziMjzuBV3IC8ODgXPeG2yrPc93JzkYJQqGUiQBIDeEXACDJv5pUT4fqwuUN739/8i4OLofdEu1kZ6MEoVDKRACkhrIHAIAkf0b0ehr0EO7Z67V1mctht2SlDZmWIBRKmQiA1BB+AQBzcl2Tmmi3dO5zU+nZGww6h93i6GxtigraUnZ3srNVNwzAX5Q9AAB842m3NFnPXmOcx4YGqbfXdTJbrrsr0LoMKE7s/AIAfONpt7S+3unkEE9Dg7PT63EUcS53sv0oEwGQfYRfAIBvvjz9nH79K5/QGROvSpJeCSzW51tv0iWfunX+op6ehTW/wWDCXd58oXUZUHwoewAA+MOydNFn/kJnTrwqI8lIOnPiVd39+JfV9twP568LhZyg29DglDgkKdWhk7YAACAASURBVG8AgFQYO7ZnYg61tLTY/f39vn0eAKCANDYmLmcYHPTzbgCUOGPMAdu2W2LX2fkFAPgj0UG2ZIfcACBLCL8AAH+49ONN+hoAZBHhF0DJ8zRRDEmlNJmtosJ5tKz513p6pOrqhV9zyimuvXoBINvo9gCgpHmaKIak0prMNjTkPJecw2rhA2u33iodP+78eelSaetWXw6z9Q2M0JYMAAfeAJS2tZv3xe0rW1cT0P6uK/JwR8XJ0/fR7UBbxGG2fAXQ2PAuOQMpsjn0AkBh4cAbgLLkaaIYkspoMtvsejiAjoxNyNb87rEfZSiJxioDKC+EXwAlLWpymId1xOfp++h2aG12PZ8BlL8EAQgj/AIoaZ2tTQpUV0atBaor1dnalKc7Kk6drU269sg/6+kdN+g/7363nt5xg6498s/R38eeHmcSW6RgcO4wWz4DKH8JAhBG+AVQ0tqa67Rp/SrV1QRk5NSoUueZurbnfqjNT2zTuSdGVSFb554Y1eYntqU0mS2fAZS/BAEI48AbACA5D4fZksn3oTO6PQDlxe3AG63OAADzLCt+K7Ikh9m8CAfNfAXQtuY6wi4Adn4BALMsS/rgB6XXX49er66WliyZD8SRUtj5BQA/sfMLAEisu3th8JWkyUnnMRicH2ARfl5Ak9koawDgBQfeAACORCUML7+c8DBbvuWzhzCA4kL4BQA43Pr0hl8LhZwSh5kZ57FAgq/EEAsA3hF+AaDcWJbTvaGiwnm0LGe9p0c65ZSF11dXF1R5QzwMsQDgFTW/AFBOLEtqb5+v3R0acp5L8zu58bo9eNzlzVfdbW1NQCNxgi5DLADEotsDAJQg1xCahX69iT4zX318891DGEDhodsDAJSo2KD7zjcv18mdu/TIvgdVe+KYji5Zpi//5APSp25VWxb69bpJVHeb6wCa7x7CAIoH4RcAiljsjufI2IRe/Zud2vTENgWnXpMknXtiVJ/97lf0hVOq1FZfH3/nN9FhN4/yXXfLEAsAXnDgDQCKWLzd1s6nHpoLvmHBqdd04xP3OwfXgsHoN8lSv163+lrqbgEUEnZ+AaCApHpg7OjYhK45/KRuf+qhuRKH2hOjca+tPXFs/uBad7dT6lBf7wTfLLQt62xtilt329nalPF7A0C2EH4BoEDEK2HYuPuQJLkG4A3P79ftMSUOMy7vf/KcWgUlJ+jmoEcvdbcAigHhFwAKRDoHxm7/0cIShwpJtiQTsTa1KKDglrs93Ucm7cqouwVQ6Kj5BYACkc6BseB/HY27bqSoUcRV99/nabeXMcEASh07vwBQINIa1ODWvSFB395EO7v5bFcGAH5g5xcACkRna5MC1ZVRa0kPjKXYvSHZzm6+25UBQK4RfgGgQLQ112nT+lWqqwnISPrA8/t14G9uVNuFb3Qms1nWwi8KhaTe3qgSB/X2upY4JNrZlWhXBqD0UfYAAAWk7bkfqu3ebqeUwRgpPIJ+aEhqb3f+HBtsU+jekGxnl3ZlAEod4RcAsiSTLgmSnJ3d9nZpfNx5Hg6+YePjTn/eDNqUJasrpl0ZgFJn7Nj/uOZQS0uL3d/f79vnAYBfYnv0Sk7HhdDb63VX2ypvb9LYGP/wWiRjpBm3Tr7p3WegulKb1q8i4AIoKcaYA7Ztt8SuU/MLAFkQr5bWlmT967D3NmHDw8mvqa9P/eYixNYV19UECL4AygplDwCQBW61tLbkvU2YW9uysARdHFLBIAoA5YydXwDIgkTdEDy3CYvXtszMzmlL0sUBAOAN4RcAsqCztSlqnHAkz23C4rUt27XLOfg2OEjwBYAsIPwCQBa0Ndcp9Pb6BQF4QZswy3IOtlVUxO/dGwo5QXdmhsALADlA+AWALLmrbZW+9N417ofJOjqk665z6npte753b7zhFQCAnKDVGQD4wbJkv/86GcX5b25Dg7PLCwDIGlqdAUAejXfeET/4St5anAEAsiJp+DXGPGCMeckY89OItTONMf9kjPmP2cczcnubAFDcFr141P3FDHv3AgC887Lz+6Ckq2LWuiT9wLbtN0n6wexzAICLo0uWxV2fkfTMhz7u780AQBlLGn5t235K0ssxy38gaefsn3dKasvyfQFASbn/qhs1XnVq1NqMpF1r1um2yvPyc1MAUIbSnfB2lm3bL87++b8knZWl+wGAkrSm6xZ1jb+u2596SLUnjunokmX6wqXX67Hz3ynjdQgGACBjGY83tm3bNsa4towwxrRLapekeuraAJSyjg7p3nudNmaSdPrpzvNQSG3NdfrMRb+nx85/54Iv8zwEAwCQsXS7PfzSGHOOJM0+vuR2oW3bvbZtt9i23bJ8+fI0Pw4AClxHh7Rjx3zwlaRf/Ur6wAfm+vh++t3nK1BdGfVlC4ZgAAByKt3w+5ikDbN/3iDpO9m5HQAoUr298denpqTubknOFLhN61e5D8EAAORc0iEXxphvSLpc0jJJv5T0aUl9kv5OUr2kIUl/bNt27KG4BRhyAaBkmdjBxjGvzcz4dy8AANchF0lrfm3bfp/LS7+T8V0BQKmorJSmp+O/xnkHACgYTHgDgGxob4+/XlUl9fT4ey8AAFeEXwDIhu3bpZtvji5/OP106cEHpVAob7cFAIiWcaszAICj70Pd2lJ/rY6OTai2JqDO1iYOswFAgSH8AihJfQMj2rL3iG9BtG9gRBt3H9LEpFP3OzI2oY27D0kSARgACghlDwBKTjiIjoxNyNZ8EO0bGMnZZ27Ze2Qu+IZNTE5ry94jOftMAEDq2PkFkBK/d1TTkSiI5upej7qMKHZbBwDkBzu/ADzLx45qOvIRRN1GFDO6GAAKC+EXgGfF8qv9fATRztYmRhcDQBEg/ALwrFh+tZ+PIMroYgAoDtT8AvCstiagkThBt9B+tR8OnGnVJluW1N0tDQ87k9l6ejz36W1rriPsAkCBI/wC8KyztSmqnZdUuL/aTyuIWpYzqW183Hk+NDQ/uY1BFQBQEoxt2759WEtLi93f3+/b5wHIvmLo9pC2xkYn8MZqaJAGB/2+m9L+XgNAjhljDti23bJgnfALoNR5DpEVFVK8/yYaI83M5P5GI8QOzZCcXXbqiAHAG7fwy4E3ACUtpfZs9fXx38RtPYeKpbMGABQbwi+AkpZSiOzpkYLB6LVg0Fn3WbF01gCAYkP4BVDSUgqRoZDU2+vU+BrjPPb25uWwG0MzACA3CL8ASlrKITIUcg63zcw4j3nq8sDQDADIDcIvgJJWrCGSoRkAkBv0+QVQ0tqa61S35+/1xm2f06+NjeqlmuX6xcf/Uhc1X5XvW0uKoRkAkH2EXwClzbJ00Wf+QpqclCSdPfaSzv7MX0iNZzC4AgDKEGUPAErbrbfOBd85k5POOgCg7BB+AZQmy3Imth0/Hv91t3UAQEmj7AFAUYs7ve25H0rt7dL4eL5vDwBQYNj5BVC03Ka3jXfekTz4Ll3qyz0CAAoL4RdA0XKb3rboxaOJv/CUU6StW3N4ZwCAQkX4BVC0XKe3LVnm/kUNDdIDD9DpAQDKFOEXQNFym9J2/1U3SsFg9GIwKD38cF6ntgEA8o/wC6BodbY26doj/6ynd9yg/7z73Xp6xw269sg/a03XLVJvr7PLa4zz2NtL6AUAyNi27duHtbS02P39/b59HoASZ1mauvFDqjo5X/4wtSigqvvvI+gCQJkzxhywbbtlwTrhF0AycduJFcLY3cZGaWho4XpDg1PeAAAoW27hlz6/ABIKtxMLd1UItxOTlP8APDyc2joAoOxR8wsgIbd2Ylv2HsnTHUWor09tHQBQ9gi/ABJybSfmsu6rnp74XR16evJzPwCAgkf4BZCQWzsxt3VfhUJ0dQAApITwCyChztYmBaoro9YC1ZXqbG3K0x3FCIWcw20zM/TwBQAkxYE3AAmFD7UVZLcHAABSRPgFkFRbcx1hFwBQEih7AAAAQNkg/AIoCM/0bNN/nXGWZkyF/uuMs/RMz7Z83xIAoARR9gAg7/7fH23QhY8+NPe38bPHXtIbPvNxPSPpou4P5/PWAAAlhp1fAPllWVoREXzDApOv6Y33fC4vtwQAKF2EXwD5Y1nShg2u/yH6tbFRX28HAFD6CL8A8sOypPZ2aXra9ZKXapb7eEMAgHJAzS+A/OjulsbHXV+ekfSLj/+lzvbvjiRJfQMj9DQGgBJG+AWQH8PDri/NSHr+2ut9P+zWNzCijbsPaWLS2Y0eGZvQxt2HJIkADAAlgrIHAPlRXx9/vbJSFQ8/rF//1k5/70fOFLtw8A2bmJzWlr1HfL8XAEBuEH4B5EdPjxQMRq8Fg9LOnVIolJdbOjo2kdI6AKD4EH4B5EcoJPX2Sg0NkjHOY29v3oKvJNXWBFJaBwAUH8IvgPwJhaTBQWlmxnnMY/CVpM7WJgWqK6PWAtWV6mxtytMdAQCyjQNvADArfKiNbg8AULoIvwDyrpDai7U11xF2AaCEUfYAIH2WJTU2ShUVzqNlpfwW4fZiI2MTsjXfXqxvYCTbdwsAAOEXQJrCE9qGhiTbdh7b21MOwLQXAwD4ifALIDXh3d73v3/hhLbxcWdyWwpoLwYA8BPhF4B3HR3Sddc5u7xuEkxui4f2YgAAPxF+ASRnWdKyZdKOHU6JQyJuk9tc0F4MAOAnuj0ASCxc2xtb4hBPMOhMbksB7cUAAH4i/AJYyLKc2t3hYaeTw/R08q9paHCCbxqDKmgvBgDwC+EXyLFC6mHrSexOb7Lga4y0a1fep7MBAOAF4RfIoXAP23Arr3APW0mFG4C7u72VOEhO8L3pJoIvAKBocOANyKGi7GHrtVvD0qXOju/27bm9HwAAsojwC+RQUfawdevWUFnp7PQ2NEgPPywdO8aOLwCg6BB+gRwqyh62PT1O14ZIwaC0c6c0MyMNDhJ6AQBFi/AL5FBR9rANhaTeXmeHN7zT29tL4AUAlAQOvAE5VLQ9bEMhwi4AoCQRfoEcK8YetkXXng0AAI8IvwCiFGV7NgAAPKLmF0CUomzPBgCAR+z8AjlWbCUERdmeDQAAjwi/QAaSBdtiLCGorQloJE7QLej2bAAAeETZA5CmcLAdGZuQrflg2zcwMndNMZYQ+NmerW9gRGs379OKrse1dvO+qO8dAAC5wM4vSpIfpQaJgm34s4qxhMCv9mzFuCsOACh+hF+UHL9ClZdgW6wlBH60Z/PylwcAALKNsgeUHL9KDbyMLs5ZCYFlSY2NUkWF82hZmb1fHhTjrjgAoPgRflFy/ApVXoJtW3OdNq1fpbqagIykupqANq1fldnOpmVJ7e3S0JBk285je3vRBWAvf3kAACDbKHtAyfGr1MBrbWzWSwi6u6Xx8ei18XFnvYhGEne2NkWVp0i5O1gHAEAY4Rclx89QFS/Y5vyw3fBwausFyq+DdQAARCL8ouTkM1T5ctiuvt4pdYi3XmT8OFgHAEAkwi9KUr5ClS8dDHp6nBrfyNKHYNBZBwAACXHgDcgiXw7bhUJSb6/U0CAZ4zz29hZVvS8AAPnCzi+QRb719Q2FCLsAAKSBnV8gi+K1PzOS3vnm5fm5IQAAEIXwC2RRW3Od/vDCOpmINVvStw+MqG9gJOravoERrd28Tyu6HtfazfsWvA4AALKP8Atk2ZP/Nio7Zi12wly4K8TI2IRszXeFIAADAJBb1PwCWebl0NuWvUfU9d2/VujZJ1Rpz2jaVMhafZW2nPYxWn8BAJBDhF8gy7wcevuzR/5K1x3cM1ceUWXP6Prw864rfLnPdOR8gAcAADlG2QOQZfEOvcVOmPuTZ5+IqguWnINxf/LsE7m/wTRRqgEAKAWEXyDL2prrtGn9KtXVBGQk1dUEtGn9qqgd0kp7Ju7Xuq0XgkQDPAAAKBaUPQA5EDVhrqNDuqhXmp6WKiul9naZykrneQxTWblgrVD4MsADAIAcI/wCufS7vyv94Afzz6enpR07pPPOk557buH17e2e3jYftbe+DfAAACCHKHsAcqWjIzr4RjpyRLr5ZmcnWHIeb75Z2r496dvmq/bWSy0zAACFzth2bEfSFL7YmEFJr0qaljRl23ZLoutbWlrs/v7+tD8PyDfPO64dHc4ObyJp/ru3dvO+uDuwdTUB7c9xpwi6PQAAioUx5kC8bJqNsod32rZ9LAvvA+RcJuEtvOMaPvQV3nGVFP0eXoJvBrW9+ay9japlBgCgCFH2gLKRabnAnY8dTt7twLKke+9N/mYea3vjcauxpfYWAIDkMg2/tqTvGWMOGGPi/q+5MabdGNNvjOkfHR3N8OOA9GXSqqtvYERjE5NxX4vace3uTljOYEv65dsu8VTb64baWwAA0pdp+L3Etu23Svp9SbcYYy6NvcC27V7btlts225Zvnx5hh8HpC+TcgG3gHzN4Sf14699UKqokBobpaEh1/ewJT20Zp3Wv+czXm7XlZc+wgAAIL6Man5t2x6ZfXzJGPP3kt4m6als3BiQbZm06ooXkK85/KQ2P7FNwanXnIWhIcmYuDu/4eD76dYOmSzU5lJ7CwBAetLe+TXGnGaMWRz+s6Tfk/TTbN0YkG2ZlAvEC8i3P/XQfPANs20nAEeY0XzwdXsvAADgj0x2fs+S9PfG+R/6Kkl/a9v2E1m5KyAHwjul6XR76Gxt0sbdh3Tlwe/r09/v1ZknX3W/2LalhgbZw8M6umS57n7HdXrs/HdKojYXAIB8y6jPb6ro84ti9v/+aIP+16MPySS7sKFBGhyURF9cAADyJZd9foHSZ1n69W/vSn5dMCj19Mw9pTYXAIDCQvhFQuxczkrSwkySs+Pb0yOFQv7cEwAASBnhF648TzQrB8PDiV+PKHUAAACFiwlvcJXJUIiSU1/v/topp0SVOgAAgMJF+IWrTIZCFK2ODqmqymlXVlXlPJeccBsMLrz+tNOkBx6g1AEAgCJB2UOBKoRa20yGQhSl3/1d6Qc/mH8+PS3t2OH8OTyOuLvbKYGor6e+FwCAIsTObwEK19qOjE3I1nytbd/AiK/3kclQiKJjWdHBN1Jvr/MYCjl1vTMzziPBFwCAokP4LUCFUmvb1lynTetXqa4mICOpriagTetXld5hN8uSNmxwfdmennZ9DQAAFBfKHgpQIdXalnyfWsuS2tudEgcX06aCf1EAACgR/G96ASq2WttCqE9OW3e3ND7u+rIt6W9XX6Xrc/DRRf19AwCgSBF+C1Bna1NUf12pcGtti74XcIL+vbakH9Wv1tfe+7Go8Os1tCa6rui/b7MI8ACAYkPNbwEqplrbQqlP9syypMZGqaLCeTzzzLiXTZkK3Xr1x/Rn12+O+kuH18OIya4ruu9bHIVyMBMAgFSw81ugiqXWtpDqk6UkO5Hh+t5wmcPQkFRd7QypeP31ufeYqD5VXa0f1oG167QpZiczUWhN5bpC+76lw+v3AgCAQkL4RUYKqT45aSlBvPreyUlp6VLp9NPn+vcGenq01aWNmdfQmuy6Qvq+pasUAjwAoPxQ9oCM+N0LuG9gRGs379OKrse1dvO+qF+xJy0lcKvvffllz/173cJp7Hqy60qhh7LX7wUAAIWE8IuM+FmfnKzGNHLH8ZrDT+rpHTfoP+9+tx7Z9L+dkof6+vhv7LYeh9fQmuy6YqrrdlMKAR4AUH4oe0DK4tXV7u+6Iuefm6zGtLYmoAv379GdP+jVGROvysxec+6JUafWd8MGaefO6NKHYNAZU+xROJwm63Dg5bpiqet24/V7AQBAITG2bfv2YS0tLXZ/f79vn4fsi62rlZzdPj92LVd0Pa54/99qJD2/+V16pmebVn7m4wpMvhb/DRoanKDb3T1X36ueHsYUAwBQgowxB2zbboldZ+cXKcnnCf9kh8Quuu8eyS34Sk7gDYUWhF161QIAUD6o+UVK8nnCP2mNaYKBFZLi1vbSqxYAgPJC+EVK8nnCP+khsUQH11xqe912sj/2d88SgAEAKEGUPSAl+R69nPCQWE9P9BCLsKVLpa1b49b2uu1YT9u2PvrIQfUPvay72lZletsAAKBAsPOLlBR0i65QSOrtdQ62GeM8PvywdOyY66G2RDvWtiTrX4fZAQYAoITQ7aFElOyhLcuK7s6wbp20Z0/WujXE614Rq64m4EsrNwAAkD10eyhhScf6FquODmnHjvnnQ0MLn7e3O39OMwCHvz8f+7tnNe3yF0HG9QIAUDooeygBScf6FiPLig66bsbHnZ3hDLQ11+mv/nj13FCMWIzrBQCgdLDzWwLy2X4skYxKMVIJtMlanHnQ1lyn/qGXZf3rcNQgDcb1AgBQWgi/BSjV0Jhs+EM+9A2M6OnPbtUj+x5U7YljOrpkmb78kw9In7rVWwBOJdAmanGWgrvaVqml4czSrJ0GAACSCL8FJ5363Xy3H4vn4Oav6rPf/YqCU87EtXNPjOqz3/2KvnBKldoe+XzyN6ivd2p6k3Hp35uuhK3UAABA0aPmt8Akqt/tGxjR2s37tKLrca3dvG+uBVde2491dEgVFU5rMWOkxYsly9KNT9w/F3zDglOv6cYn7vf2vj09es1ULlielvRyYPF8K7Pe3oy6PQAAgPJCq7MCs6Lrcbn9RALVlQt2d30LuZEtx84801k7fjz+tVVVsqem4h4gm5FRhT3j6SPvfO8n9OePbdOZJ1+VJL0SWKw7f6ddB9auo/UYAABIiFZnRcKtfrfSGNcd4ZyFX8uSbr11Ych1C71hU1OyKyplZhb2zj15Tq2CHj9+TdctuuQ3L10Y+DmABgAA0kTZQ4HpbG1SoDr61/2B6kr/e9BalvTBDyYPui4qZqY1tSj6wN3UooCCW+72/B4FPU0OAAAUJcJvgXELfHUunRvS6uhgWVJjo1Or29joPI/V3S29/nrq7x3W0KCq+++LGjVcdf99KdfntjXXaX/XFXp+87u0v+sKgi8AAMgIZQ8FyK3jQFodHeKNB9650xkOIblPScukd25V1fzYYQ6jAQCAAsKBtyKS8tAIy3KCbTjoSs4ubLyfeUODNDg4/7yx0VursVinny7de29WQm9GQzIAAEBZczvwRvgtFR0dTtuv6WmpstIJvXv2eA+wxkgzEV0YwjW/8UofTj9dOvVU6eWXnd3k8C5vFsX2O5Z87m4BAACKmlv4pea3FHR0yN6xwwm+kjQ97TxPZec2dkpaKCQ98IC0dOn82tKl0sMPS6++Kh075oTlwcGclDYk6ncMAACQLmp+S8DM13oX/C3GSLJnHxeILX1wm5KWx5pdty4W8drAAQAAeMXObwmI1093TjC48PlNN0V1YSjEKWluXSyMNDfZDgAAIFWE3xIwbeL/GKdNhRNsY4Ju34e6tfamB7Ti9n/Q2pseUN95l/t7wx50tjbF3bW2JUofAABA2gi/xcSlP+933nb1gpHI9uy6QiGnLne2PrfvvMu1cfchjYxNyJZTRrBx96GC201ta65zHfOcs8EeAACg5BF+i0W4bdnQkFOvG+7Pa1mq3LFdf3vhuzRlKmRLmjIV+tsL36XKHdsXvE0qB8n6Bka0dvM+reh6XGs37/M9IGd1sAcAAIA48FYcLEvasGG+m0PY+LjU3a22wUH13fc1XeahJ67brmnsemyrsfAOsSTfWo11tjalN9gDAADABeG30IV3fGODb9jsJDa3qXCxamsCcTsmxO6mJtoh9iv8hj+HQRcAACBbCL+Frrs7ekJbrNj+vEl43U31ukOca15DPQAAgBeE3wLhOsp3dmc3Lrf+vAl43U31ukMMAABQTAi/BSBefe3Tn92q3/vJ3yroNn66sjLt/rxedlOptwUAAKWI8FsAYutrP7N3u647uMe9FUcwmPPBFNTbAgCAUkT4LQCRdbTXHH4ycfBtaHBKHXyYyEa9LQAAKDWE3wIQWV97+1MPuQdfY5yBFSlwrSUGAAAoQ4TfPIgNpO9883Kd3LlLt+17UHUnRt2/MMXODoXQqxcAAKCQMOHNZ+FAGjle+OTOXfr8P/61zj0xKuP2hcak3NkhlWluAAAA5YDw67N4gfS2fQ/qlNdOun+RMdJNN6Vc51sovXoBAAAKBeHXZ/GCZ+2JY+5f0NAg7dolbd+e8me59eSlVy8AAChXhF+fxQueR5csi39xQ4NzwC3Nzg6drU0KVFdGrdGrFwAAlDPCr8/iBdIvX/EBTS2KCcVpTG+L1dZcp03rV6muJiAjqa4moE3rV3HYDQAAlC1ju00Qy4GWlha7v7/ft88rVM/0bNMb7/mcfm1sVC/VLNcvPv6XuqjxDKm72xlnXF/vWy/fTNFKDQAAFCJjzAHbtlti12l15jfL0kWfv0MaH5cknT32ks7+/B3OxLYUe/jmG63UAABAsaHswW/d3XPBd874uLNeZGilBgAAig3h12/Dw6mtFzBaqQEAgGJD+PWb25S2FKe3FQJaqQEAgGJD+PVbT4/TySFSFjo75AOt1AAAQLEh/OaaZUmNjVJFhfMoOYfbGhqcyW0NDc7zIujsEItWagAAoNjQ6iyXLEtqb48+4BYMFm3YBQAAKBZurc7Y+c2lEursAAAAUAro85uBpAMeSqizAwAAQClg5zdN4QEPI2MTsjU/4KFvYGT+ohLq7AAAAFAKCL9p8jTgoYQ6OwAAAJQCwm+aPA14CIVKprMDAABAKaDmN0XhOt/YHhnXHH5Sd/6gV2dMvCrdLWnpUmnrVifoEnYBAAAKAuE3BeE639hyh2sOP6l7/nGrTpmeml88fly64QY9M/iKbqs8z/1QXJFLeugPAACggBB+UxCvzleSbn/qoejgGzY5qdotn9PITQ9Imj8UJ6noAmK8kCsp6i8DxfzPBwAAygM1vylwq/OtPXHM9WvO+e/RqOcLDsUVAbfOFnc+djj5oT8AAIACws5vCmprAhqJCMDXHH5Stz/1kMyCCuB5R5csW7jmEqLd5Lu0wK2zRbxdcCn1fz4AAAC/sPObgs7WJgWqKyU5wXfzE9t0M7NFEgAACgNJREFU7olRGZfrX6+o1BcuvX7Bem1NwPNneuonnGOphtlU/vkAAAD8RPhNQVtznTatX6W6moBuf+ohBadec7946VI9+9kv65/W/G7UcqC6cq5e1gtP/YRzzC3MnhGsnvvLQFiq/3wAAAB+IvymqK25Tvu7rtC5r7rU+Roj2bZ07Jgu6v7wXFg2kupqAtq0flVKJQue+gnnWOSOd1igulKffvf5Gf/zAQAA+Ima31RZltTd7QTceGJGF7c112UUBmPrjCPX/RK+f7e6Y8IuAAAoFoTfJCIPm214fr8++diXVXUy/q7r1KKA7vqtP9HOrsezdjCts7VpQW/hfJQWZBriAQAACgHhN4HYoRY3PnG/a/AdP6dOn/qtP9GjK9ZKyl7P22S7rgAAAPCubMOvl/ZhsYfNXPv5GqMrP/LQgvKE8MG0TIMqu64AAADZUZbhN3ZH122X9ujYxFwv39oTxzRjjCri1frW1xfEwTQAAAAkVpbdHry2D9vw/P65Xr4VslVlzywcZxEMSj09rgfQ6HkLAABQOMoy/Hrdpb39Rwt7+RpJMxWVTkuzhgapt1cKhVzbgdHzFgAAoHCUZdmDp/ZhlqXgi/GnqFXYM9LMzIL1RdUVczvKNYFq3XnN+dTqAgAAFJCy3PlNuktrWZq68UPubxDTyzdcQ/zK+OTc2mtTC8MxAAAA8qssw2/kmOJ4k8nGO+9wbWkWrvGNVAgjiAEAAJBcWZY9SInbhy168WjcdVuSma3xjUSnBwAAgOJQlju/yRxdsizu+siS5QuCr+Te0YFODwAAAIWF8BvH/VfdqPGqU6PWxqtO1f1X3Rj3ejo9AAAAFIeMwq8x5ipjzBFjzM+NMV3Zuql8W9N1iz519Uf0wpLlmpHRC0uW61NXf0Rrum6Je32yGmIAAAAUBmPHm1jm5QuNqZT075KulPSCpGckvc+27efcvqalpcXu7+9P6/P85mX8MQAAAAqTMeaAbdstseuZHHh7m6Sf27b9n7Mf8E1JfyDJNfwWk0QH4gAAAFCcMil7qJP0i4jnL8yuRTHGtBtj+o0x/aOjoxl8XJZZltTYKFVUOI+Wle87AgAAQI7l/MCbbdu9tm232Lbdsnz58lx/nDeWJbW3S0NDkm07j+3tBGAAAIASl0n4HZH0xojn586uFb7ubml8PHptfNxZBwAAQMnKJPw+I+lNxpgVxphTJP1vSY9l57ZyyLKcnd54hof9vRcAAAD4Ku0Db7ZtTxljPixpr6RKSQ/Ytn04a3eWC+FyBzf19f7dCwAAAHyX0Xhj27b3SNqTpXvJvXjlDmHBoNTT4+/9AAAAwFflNeEtUVlDb2/c0cUAAAAoHeUVft3KGhoaCL4AAABloLTDb2wv33XrnPKGSJQ7AAAAlI3SDb/xevnu3Clt2ODs9BrjPFLuAAAAUDYyOvBW0Nx6+e7ZIw0O5uWWAAAAkF+lu/PrdriNXr4AAABlq3R3fuvr4w6zGD+7Vldu3qejYxOqrQmos7VJbc11ebhBAAAA+K10d357ehYcbptaFNCnfutPNDI2IVvSyNiENu4+pL6B4pjKDAAAgMyUbvgNhZzDbBGH2+665jY92nRZ1GUTk9PasvdInm4SAAAAfird8Cs5AXhwUJqZkQYHtXPF2riXHR2b8Pe+AAAAkBelHX5j1NYEUloHAABAaSmr8NvZ2qRAdWXUWqC6Up2tTXm6IwAAAPipdLs9xBHu6rBl7xG6PQAAAJShsgq/khOACbsAAADlqazKHgAAAFDeSnrnt29ghBIHAAAAzCnZ8Ns3MKKNuw9pYnJa0vxAC0kEYAAAgDJVsmUPW/YemQu+YQy0AAAAKG8lG37dBlcw0AIAAKB8lWz4ZaAFAAAAYpVs+GWgBQAAAGKV7IE3BloAAAAgVsmGX4mBFgAAAIhWsmUPAAAAQCzCLwAAAMoG4RcAAABlg/ALAACAskH4BQAAQNkg/AIAAKBsEH4BAABQNgi/AAAAKBuEXwAAAJQNwi8AAADKBuEXAAAAZYPwCwAAgLJB+AUAAEDZIPwCAACgbBB+AQAAUDYIvwAAACgbhF8AAACUDWPbtn8fZsyopKEcfsQyScdy+P4oHPysywc/6/LBz7p88LMuH/n8WTfYtr08dtHX8Jtrxph+27Zb8n0fyD1+1uWDn3X54GddPvhZl49C/FlT9gAAAICyQfgFAABA2Si18Nub7xuAb/hZlw9+1uWDn3X54GddPgruZ11SNb8AAABAIqW28wsAAAC4Kpnwa4y5yhhzxBjzc2NMV77vB7lhjHmjMeZJY8xzxpjDxphb831PyB1jTKUxZsAY89183wtyyxhTY4x51Bjzb8aYnxljLs73PSE3jDEfnf3v90+NMd8wxizK9z0hO4wxDxhjXjLG/DRi7UxjzD/9//buJ0SLOgDj+PfBNdANRBDEdgU9iLIEoogsLUi0HpTC7SQGiYTH/igEol66doiog3SxVFAKMUEPUYEdvImkQZSHYq3dtfUPhH/wsi49HWYCEUGod/ZnM8/nMr/5nR4Y3vd93pnfzEj6pd4uLpkRWlJ+Jc0DDgFbgSHgNUlDZVNFQ2aBd20PAcPAmznWrbYHuFI6RMyJj4Gvba8B1pLj3kqSBoB3gA22nwfmATvKpooeOgpseWRuP3DO9irgXL1fVCvKL7AR+NX2uO0Z4AtgrHCmaIDtaduX6vE9qh/IgbKpogmSBoGXgcOls0SzJC0CNgGfAtiesX27bKpoUB+wQFIfsBD4o3Ce6BHb54E/H5keA47V42PAq3Ma6jHaUn4HgMmH9qdIIWo9SSuAdcCFskmiIR8B+4C/SgeJxq0EbgFH6mUuhyX1lw4VvWf7GvABMAFMA3dsf1s2VTRsqe3penwdWFoyDLSn/EbHSHoW+BLYa/tu6TzRW5JeAW7a/r50lpgTfcB64BPb64D7PAWXRqP36vWeY1R/eJ4D+iW9XjZVzBVXjxgr/pixtpTfa8Dyh/YH67loIUnzqYrvCdunS+eJRowA2yT9RrWM6SVJx8tGigZNAVO2/7mKc4qqDEf7bAau2r5l+wFwGnihcKZo1g1JywDq7c3CeVpTfi8CqyStlPQM1eL5s4UzRQMkiWpd4BXbH5bOE82wfcD2oO0VVJ/n72zn7FBL2b4OTEpaXU+NAj8XjBTNmQCGJS2sv89Hyc2NbXcW2FWPdwFnCmYBqktN/3u2ZyW9BXxDdefoZ7Z/KhwrmjEC7AR+lPRDPXfQ9lcFM0XEf/c2cKI+gTEOvFE4TzTA9gVJp4BLVE/vucxT+Aaw+HckfQ68CCyRNAW8B7wPnJS0G/gd2F4uYSVveIuIiIiIzmjLsoeIiIiIiCdK+Y2IiIiIzkj5jYiIiIjOSPmNiIiIiM5I+Y2IiIiIzkj5jYiIiIjOSPmNiIiIiM5I+Y2IiIiIzvgbK3fqCQ58vTgAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "plt.figure(figsize=(12, 9))\n",
        "plt.scatter(x_samples.flatten(), y_samples.flatten())\n",
        "y_prediction = model.apply(trained_variables, x_samples)\n",
        "plt.scatter(x_samples.flatten(), y_prediction.flatten(), c='red', label='lineGrid Prediction')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUpDh2EEsEDh"
      },
      "source": [
        "## ImageGrid\n",
        "\n",
        "`ImageGrid` is a grid of feature vectors arranged in a 2D grid. For a $w\\times h=4\\times 3$ image, we store those on the *corners* of the pixels, so we need storage of shape $(h+1,w+1, d)$, with $d$ the feature dimension. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "mCO0WFSePqi9"
      },
      "outputs": [],
      "source": [
        "class ImageGrid(nn.Module):\n",
        "  width: int\n",
        "  height: int\n",
        "  features: int = 1\n",
        "  grid_init: Initializer = nn.initializers.lecun_normal()\n",
        "\n",
        "  @staticmethod\n",
        "  def interpolate(feature_grid, x, y):\n",
        "    \"\"\"Interpolate vectors on a regular 2D grid.\"\"\"\n",
        "    (xd, yd), whole = jnp.modf(jnp.stack([x, y]))\n",
        "    x0, y0 = whole.astype(int)\n",
        "    x1, y1 = x0 + 1, y0 + 1\n",
        "\n",
        "    def f(grid):\n",
        "      # Interpolate along x\n",
        "      c0 = grid[y0, x0] * (1 - xd) + grid[y0, x1] * xd\n",
        "      c1 = grid[y1, x0] * (1 - xd) + grid[y1, x1] * xd\n",
        "\n",
        "      # Interpolate along y\n",
        "      return c0 * (1 - yd) + c1 * yd\n",
        "\n",
        "    return jax.vmap(f, -1, -1)(feature_grid)\n",
        "\n",
        "  @nn.compact\n",
        "  def __call__(self, p):\n",
        "    grid = self.param('grid', self.grid_init,\n",
        "                      (self.height + 1, self.width + 1, self.features))\n",
        "    return self.interpolate(grid, p[..., 0], p[..., 1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XRESLylXnhxY"
      },
      "source": [
        "One application of the ImageGrid would be to train a \"sky image\", i.e., fit a panorama at a distance to store a direction-dependent background color."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z60ukTmakvxr"
      },
      "source": [
        "## VoxelGrid\n",
        "\n",
        "For a Voxel-based Radiance Field, we need a *3D* voxel volume. The `VoxelGrid` module below uses feature vectors arranged in a 3D grid. For a $w\\times h \\times d$ voxel grid, we store those on the *corners* of the voxels, so we need storage of shape $(d+1,h+1,w+1, d)$."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "yuaPSkw2lENw"
      },
      "outputs": [],
      "source": [
        "class VoxelGrid(nn.Module):\n",
        "  width: int\n",
        "  height: int\n",
        "  depth: int\n",
        "  features: int = 1\n",
        "  grid_init: Initializer = nn.initializers.lecun_normal()\n",
        "\n",
        "  @staticmethod\n",
        "  def interpolate(feature_grid, x, y, z):\n",
        "    \"\"\"Interpolate vectors on a regular 3D grid.\"\"\"\n",
        "    (xd, yd, zd), whole = jnp.modf(jnp.stack([x, y, z]))\n",
        "    x0, y0, z0 = whole.astype(int)\n",
        "    x1, y1, z1 = x0 + 1, y0 + 1, z0 + 1\n",
        "\n",
        "    def f(grid):\n",
        "      # Interpolate along x\n",
        "      c00 = grid[z0, y0, x0] * (1 - xd) + grid[z0, y0, x1] * xd\n",
        "      c01 = grid[z0, y1, x0] * (1 - xd) + grid[z0, y1, x1] * xd\n",
        "      c10 = grid[z1, y0, x0] * (1 - xd) + grid[z1, y0, x1] * xd\n",
        "      c11 = grid[z1, y1, x0] * (1 - xd) + grid[z1, y1, x1] * xd\n",
        "\n",
        "      # Interpolate along y\n",
        "      c0 = c00 * (1 - yd) + c01 * yd\n",
        "      c1 = c10 * (1 - yd) + c11 * yd\n",
        "\n",
        "      # Interpolate along z\n",
        "      return c0 * (1 - zd) + c1 * zd\n",
        "\n",
        "    return jax.vmap(f, -1, -1)(feature_grid)\n",
        "\n",
        "  @nn.compact\n",
        "  def __call__(self, p):\n",
        "    grid = self.param(\n",
        "        'grid', self.grid_init,\n",
        "        (self.depth + 1, self.height + 1, self.width + 1, self.features))\n",
        "    return self.interpolate(grid, p[..., 0], p[..., 1], p[..., 2])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "chvC8B0lnwdA"
      },
      "source": [
        "## Training a VoxelGrid\n",
        "\n",
        "Directly supervising a 3D field of feature vectors is just as simple as in the 1D (or 2D) case. Because we redefine the `model` variable below as a `VoxelGrid` instance, we can reuse the `train` function as is!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y177fXT3BapT",
        "outputId": "ac4e06b1-1ebc-4276-8687-3cf28f0ce17d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss step 0:  11.062546\n",
            "Loss step 500:  0.32209575\n",
            "Loss step 1000:  0.075519726\n",
            "Loss step 1500:  0.029905703\n",
            "Loss step 2000:  0.015100392\n"
          ]
        }
      ],
      "source": [
        "# Generate expected grid and ground truth\n",
        "W, H, D = 4, 3, 2\n",
        "xx, yy, zz = np.meshgrid(np.arange(D+1), np.arange(H+1), np.arange(W+1), indexing='ij')\n",
        "true_grid = xx + yy + zz\n",
        "true_grid = jnp.expand_dims(true_grid, axis=3)\n",
        "true_variables = freeze({'params': {'grid': true_grid}})\n",
        "\n",
        "# Generate training data\n",
        "num_x = 4096\n",
        "x_samples = np.random.uniform(size=(num_x, 3))\n",
        "for i, size in enumerate([W,H,D]):\n",
        "  x_samples[:, i] += np.random.randint(0, size, size=(num_x))\n",
        "y_samples = VoxelGrid.interpolate(true_grid, x_samples[:, 0], x_samples[:, 1], x_samples[:, 2])\n",
        "\n",
        "# Setup model and initialize parameters\n",
        "model = VoxelGrid(W, H, D)\n",
        "variables = model.init(random.PRNGKey(42), x_samples)\n",
        "\n",
        "trained_variables = train(variables, x_samples, y_samples, learning_rate=0.3, num_epochs=2500, checkpoint_freq=500)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZVrnzjCCineD"
      },
      "source": [
        "## DVGO\n",
        "\n",
        "For NeRF, we need to add volume rendering. We also need a bounding box scaling module to transform points from scene coordinates to grid coordinates:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "BSal--qhd0r1"
      },
      "outputs": [],
      "source": [
        "@dataclasses.dataclass\n",
        "class BBox(object):\n",
        "  \"\"\"Scaling coordinates to a bbox.\"\"\"\n",
        "  min_corner: Tuple[float, float, float] = (-1, -1, -1)\n",
        "  max_corner: Tuple[float, float, float] = (1, 1, 1)\n",
        "\n",
        "  def grid_from_scene(self, p):\n",
        "    \"\"\"Scale scene coordinates to axis-aligned bounding box.\"\"\"\n",
        "    min_corner = jnp.array(self.min_corner, float)\n",
        "    max_corner = jnp.array(self.max_corner, float)\n",
        "    scale = 1.0 / (max_corner - min_corner)\n",
        "    unclamped = (p - min_corner) * scale\n",
        "    return jnp.clip(unclamped, 0.0, 1.0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "grnqYchZd36n"
      },
      "source": [
        "We set up a Renderer class that we will inherit from as well: this provides the functionality to \n",
        "- sample 3D points along a ray (`sample_along_ray`)\n",
        "- to take density and color values at those samples and perform volume rendering (`render`)\n",
        "\n",
        "Note the Renderer also has a provision for a background color: if the total opacity along the ray *inside* the grid does not add up to 1, we use the remaining opacity to render the background color."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "OeMdUwmyeGax"
      },
      "outputs": [],
      "source": [
        "@dataclasses.dataclass\n",
        "class Renderer(object):\n",
        "  \"\"\"Just the volume rendering code.\"\"\"\n",
        "  near: float = 0.0\n",
        "  far: float = 1.0\n",
        "  num_samples: int = 32\n",
        "  background: Tuple[float, float, float] = (0., 0., 0.)\n",
        "\n",
        "  def sample_along_ray(self, rng, ray_origin, ray_direction, randomized=False):\n",
        "    \"\"\"Sample along the ray.\"\"\"\n",
        "    t_vals = jnp.linspace(0., 1., self.num_samples + 1)\n",
        "    t_vals = self.near * (1. - t_vals) + self.far * t_vals\n",
        "    t_mids = 0.5 * (t_vals[1:] + t_vals[:-1])\n",
        "\n",
        "    if randomized:\n",
        "      upper = jnp.concatenate([t_mids, t_vals[-1:]], -1)\n",
        "      lower = jnp.concatenate([t_vals[:1], t_mids], -1)\n",
        "      t_rand = random.uniform(rng, [self.num_samples + 1])\n",
        "      t_vals = lower + (upper - lower) * t_rand\n",
        "\n",
        "    samples = ray_origin + jnp.expand_dims(t_mids, 1) * ray_direction\n",
        "    return t_vals, samples\n",
        "\n",
        "  def render(self, t_vals, ray_direction, density, rgb):\n",
        "    \"\"\"Volumetric Rendering Function.\"\"\"\n",
        "    t_dists = t_vals[1:] - t_vals[:-1]\n",
        "    delta = t_dists * jnp.linalg.norm(ray_direction)\n",
        "    density_delta = density * delta\n",
        "\n",
        "    alpha = 1 - jnp.exp(-density_delta)\n",
        "    trans = jnp.exp(-jnp.concatenate(\n",
        "        [jnp.zeros_like(density_delta[:1]),\n",
        "         jnp.cumsum(density_delta[:-1])]))\n",
        "    weights = alpha * trans\n",
        "\n",
        "    acc = weights.sum()\n",
        "    return weights @ rgb + (1.0 - acc) * jnp.array(self.background)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKpSY_xdeKhH"
      },
      "source": [
        "Finally we provide the trainable density and color grids as two VoxelGrid instances. They just add a specific initializer function, and a non-linearity to make sure the output densitites and colors are scaled appropriately:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "jpxwYpmRvORf"
      },
      "outputs": [],
      "source": [
        "class DensityGrid(nn.Module):\n",
        "  \"\"\"Interpolate a 1D field and softplus it for density.\"\"\"\n",
        "  shape: Tuple[int, int, int]  # W, H, D\n",
        "\n",
        "  @nn.compact\n",
        "  def __call__(self, p):\n",
        "    raw = VoxelGrid(\n",
        "        *self.shape,\n",
        "        features=1,\n",
        "        grid_init=nn.initializers.constant(0.005),\n",
        "        name='grid')(p)\n",
        "    return nn.softplus(jnp.squeeze(raw))\n",
        "\n",
        "\n",
        "class RGBGrid(nn.Module):\n",
        "  \"\"\"Interpolate a 3D (RGB) field and restrict to [0,1] with sigmoid.\"\"\"\n",
        "  shape: Tuple[int, int, int]  # W, H, D\n",
        "\n",
        "  @nn.compact\n",
        "  def __call__(self, p):\n",
        "    raw = VoxelGrid(\n",
        "        *self.shape,\n",
        "        features=3,\n",
        "        grid_init=nn.initializers.constant(0.0),\n",
        "        name='grid')(p)\n",
        "    return nn.sigmoid(raw)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gTaoIdL7eZwm"
      },
      "source": [
        "Now we are ready to create the Simplified Differentiable Voxel Grid, or `SimpleDVGO`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "GLHsfwnRZ3kM"
      },
      "outputs": [],
      "source": [
        "class SimpleDVGO(nn.Module, Renderer, BBox):\n",
        "  \"\"\"Simplified Differentiable Voxel Grid, see Sun et al. CVPR 2020.\"\"\"\n",
        "  shape: Tuple[int, int, int] = (128, 128, 128)  # W, H, D\n",
        "\n",
        "  @nn.compact\n",
        "  def __call__(self, rng, example, randomized=False):\n",
        "    \"\"\"Volume Rendering.\"\"\"\n",
        "    t_vals, samples = self.sample_along_ray(rng, example['ray_origins'],\n",
        "                                            example['ray_directions'],\n",
        "                                            randomized)\n",
        "    rescaled = self.grid_from_scene(samples) * jnp.array([self.shape])\n",
        "    density = DensityGrid(shape=self.shape, name='density')(rescaled)\n",
        "    rgb = RGBGrid(shape=self.shape, name='color')(rescaled)\n",
        "    return self.render(t_vals, example['ray_directions'], density, rgb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0C0hvf5vqV5t"
      },
      "source": [
        "## Examining the Scene\n",
        "We will use the synthetic `lego` scene from the original NeRF paper. This dataset contains 100 `frame` instances, each with their own pose $T^s_i$ that converts from frame $i$ coordinates to scene coordinates.\n",
        "\n",
        "Below we load the dataset using Google's [sunds](https://github.com/google-research/sunds) library. By specifying the `Scenes` task we get a description of the data rather than the data itself:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "1mNnS0LIogZx"
      },
      "outputs": [],
      "source": [
        "scene_ds = sunds.load('nerf_synthetic/lego', \n",
        "                      split='train',\n",
        "                      task=sunds.tasks.Scenes())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sV4dviY9yrm_"
      },
      "source": [
        "There is only one element, which is a dictionary with scene information:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n5elltf8wz-q",
        "outputId": "0fdbb2f1-6158-49af-d7b2-794eb18b01d9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dict_keys(['frames', 'scene_box', 'scene_name', 'timestamp'])\n",
            "scene_name = b'lego'\n",
            "timestamp = 0.0\n",
            "scene_box:\n",
            "min_corner = [-0.637787 -1.140016 -0.344656]\n",
            "max_corner = [0.633744 1.148737 1.002206]\n"
          ]
        }
      ],
      "source": [
        "scene = scene_ds.get_single_element()\n",
        "print(scene.keys())\n",
        "print(f\"scene_name = {scene['scene_name']}\")\n",
        "print(f\"timestamp = {scene['timestamp']}\")\n",
        "print(\"scene_box:\")\n",
        "print(f\"min_corner = {scene['scene_box']['min_corner'].numpy()}\")\n",
        "print(f\"max_corner = {scene['scene_box']['max_corner'].numpy()}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vsydcW7Rk86y",
        "outputId": "71b9cd03-2112-427d-c31d-8628eac8fc11"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "max_corner [0.633744 1.148737 1.002206]\n",
            "min_corner [-0.637787 -1.140016 -0.344656]\n"
          ]
        }
      ],
      "source": [
        "bounding_box = tfds.as_numpy(scene['scene_box'])\n",
        "for key, value in bounding_box.items():\n",
        "  print(key, value)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rC-gWFcYztZ1"
      },
      "source": [
        "## Examining the Frames\n",
        "\n",
        "The frames field contains a description for all frames. There are 100 frames, so let's examine the first 5 here:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XeWZz1SyydKA",
        "outputId": "b249a810-9533-407f-9ea9-04fec0a0b1ca"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Frame keys: dict_keys(['cameras', 'frame_name', 'pose', 'timestamp'])\n",
            "Number of frames: (100,)\n",
            "First 5 frame names: [b'lego_train_frame0000' b'lego_train_frame0001' b'lego_train_frame0002'\n",
            " b'lego_train_frame0003' b'lego_train_frame0004']\n",
            "First 5 time stamps: [0. 0. 0. 0. 0.]\n"
          ]
        }
      ],
      "source": [
        "frames = scene['frames']\n",
        "print(f\"Frame keys: {frames.keys()}\")\n",
        "print(f\"Number of frames: {frames['frame_name'].shape}\")\n",
        "print(f\"First 5 frame names: {frames['frame_name'].numpy()[:5]}\")\n",
        "print(f\"First 5 time stamps: {frames['timestamp'].numpy()[:5]}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QBC9y9XercaL"
      },
      "source": [
        "As you can see, in this synthetic dataset all time stamps are zero."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uqi98bjq2py_"
      },
      "source": [
        "To show the images, we load just use the `Frames` task, which -as the name suggests- yields frames as tfds example objects. With yet anoteher cool Google library, [mediapy](https://github.com/google/mediapy), we can show some images:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "OCUqRA5P24Us"
      },
      "outputs": [],
      "source": [
        "frame_ds = sunds.load('nerf_synthetic/lego', \n",
        "                      split='train',\n",
        "                      task=sunds.tasks.Frames())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gVL5uuFR_gZf"
      },
      "source": [
        "The image size is $800 \\times 800$. Below we show the first image at a quarter resolution:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        },
        "id": "Oe1y2JbH_BQl",
        "outputId": "06ae8ee7-4864-47aa-a3a8-3b599f46f26e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Image shape:  (800, 800, 3)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>lego_train_frame0007</div><div><img width=\"100\" height=\"100\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "frame0 = next(iter(frame_ds))\n",
        "image0 = frame0['cameras']['default_camera']['color_image']\n",
        "print('Image shape: ', image0.shape)\n",
        "media.show_image(image0, title=frame0['frame_name'].numpy().decode(), height=100)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "npyLHvApASKE"
      },
      "source": [
        "But we can also show *all* 100 frames:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 406
        },
        "id": "oQ4z8QYo3J2T",
        "outputId": "86399a9b-4708-4c79-bc6d-a9433c8f533d"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0007</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0075</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0040</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0079</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0091</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0019</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0028</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0011</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0057</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0013</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0027</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0085</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0082</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0096</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0071</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0089</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0045</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0058</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0035</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0015</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table><table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0041</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0044</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0090</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0080</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0020</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0037</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0026</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0093</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0039</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0059</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0049</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0033</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0003</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0017</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0074</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0038</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0005</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0031</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0088</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0056</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table><table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0052</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0021</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0077</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0032</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0053</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0043</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0054</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0076</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0061</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0022</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0098</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0034</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0067</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0068</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0050</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0078</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0046</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0024</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0000</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0004</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table><table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0062</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0099</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0094</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0012</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0087</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0006</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0073</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0029</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0083</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0070</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0009</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0002</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0081</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0030</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0018</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0072</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0008</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0036</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0025</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0063</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table><table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0042</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0064</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0086</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0023</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0016</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0051</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0084</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0048</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0066</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0097</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0055</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0047</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0060</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0001</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0065</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0095</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0069</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0092</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0014</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>0010</div><div><img width=\"50\" height=\"50\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "titles = [frame['frame_name'].numpy().decode()[-4:] for frame in frame_ds]\n",
        "images = [frame['cameras']['default_camera']['color_image'].numpy() for frame in frame_ds]\n",
        "media.show_images(images, height=50, titles=titles, columns=20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K1Ua4I7Hip4U"
      },
      "source": [
        "## Volume Rendering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Snxz-THz2S9h"
      },
      "source": [
        "The real usefulness of [sunds](https://github.com/google-research/sunds) library for NeRF-style training is that it automatically calculates the rays for use! We do this using the `Nerf` task:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bxrm_fu1ODlP",
        "outputId": "fa0e4c42-542a-4831-9d4b-561ca4c86be2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'color_image': TensorSpec(shape=(800, 800, 3), dtype=tf.uint8, name=None), 'ray_origins': TensorSpec(shape=(800, 800, 3), dtype=tf.float32, name=None), 'ray_directions': TensorSpec(shape=(800, 800, 3), dtype=tf.float32, name=None)}\n"
          ]
        }
      ],
      "source": [
        "ds = sunds.load('nerf_synthetic/lego', split='train', \n",
        "                task=sunds.tasks.Nerf(yield_mode='image', ),\n",
        "                shuffle_files=True).cache()\n",
        "print(ds.element_spec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8UswDzwkqpM7"
      },
      "source": [
        "We can use this to test the volume rendering functionality. First, we create some variables."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "I1CVyC_M6nCE"
      },
      "outputs": [],
      "source": [
        "frame0 = tfds.as_numpy(next(iter(ds)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kZSSxssmnmx-",
        "outputId": "96def022-d108-4568-e72f-885c8cabf4c8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Grid W, H, D = (95, 191, 95)\n"
          ]
        }
      ],
      "source": [
        "scale = 32*3\n",
        "W, H, D = scale * (bounding_box['max_corner']-bounding_box['min_corner']).astype(int) - 1\n",
        "print(f\"Grid W, H, D = {W, H, D}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "E36HJQ0Rac_E"
      },
      "outputs": [],
      "source": [
        "dvgo = SimpleDVGO(shape=(W, H, D), near=2.5, far=5.5, num_samples=64, **bounding_box, background=(1, 1, 1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "_H3qetNNo9uW"
      },
      "outputs": [],
      "source": [
        "color_grid = np.empty((D+1, H+1, W+1, 3))\n",
        "color_grid[..., :] = np.array([0.0, 1.0, 0.0])\n",
        "density_grid = np.full((D+1, H+1, W+1, 1), 0.0)\n",
        "density_grid[:,H//2,:,:] = 1.0\n",
        "variables = freeze({\n",
        "    'params': {\n",
        "        'color': {\n",
        "            'grid': {\n",
        "                'grid': color_grid\n",
        "            }\n",
        "        },\n",
        "        'density': {\n",
        "            'grid': {\n",
        "                'grid': density_grid\n",
        "            }\n",
        "        }\n",
        "    }\n",
        "})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "ucSamE6wMX7h"
      },
      "outputs": [],
      "source": [
        "def downsample(frame, stride):\n",
        "  return {k: v[::stride, ::stride] for k, v in frame.items()}\n",
        "\n",
        "def select(frame, i):\n",
        "  return {k: v[i] for k, v in frame.items()}\n",
        "\n",
        "def render_image(variables, frame, stride=4):\n",
        "  downsampled = downsample(frame, stride)\n",
        "  # render one scanline at a time\n",
        "  rng = random.PRNGKey(42)\n",
        "  render_rays = jax.jit(jax.vmap(lambda ray: dvgo.apply(variables, rng=rng, example=ray, randomized=False)))\n",
        "  rendered = np.stack([render_rays(select(downsampled, i))\n",
        "                       for i in range(800//stride)])\n",
        "  return rendered"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "hCzIZ15Z6NfN"
      },
      "outputs": [],
      "source": [
        "stride = 8\n",
        "rendered = render_image(variables, frame0, stride)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        },
        "id": "APjLiXvrwUPv",
        "outputId": "15a318cb-3b2b-4608-f212-d49abb14fae7"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><img width=\"100\" height=\"100\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></td></tr></table>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "media.show_image(rendered)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJV9KfepdNzk"
      },
      "source": [
        "## Batching Rays with `tfds`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "xf1g8gFHVGPm"
      },
      "outputs": [],
      "source": [
        "num_images = 100\n",
        "shuffled_images_ds = ds.shuffle(num_images)\n",
        "batched_images_ds = shuffled_images_ds.batch(num_images, num_parallel_calls=tf.data.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "UcJ8AeIQVvaI"
      },
      "outputs": [],
      "source": [
        "def sample_data(x, image_size):\n",
        "  color_image = tf.reshape(x['color_image'], shape=[-1, 3])\n",
        "  ray_origins = tf.reshape(x['ray_origins'], shape=[-1, 3])\n",
        "  ray_directions = tf.reshape(x['ray_directions'], shape=[-1, 3])\n",
        "\n",
        "  idx = tuple(np.random.choice(image_size, size=image_size, replace=False))\n",
        "\n",
        "  return {\n",
        "      'color_image': tf.gather(color_image, idx),\n",
        "      'ray_origins': tf.gather(ray_origins, idx),\n",
        "      'ray_directions': tf.gather(ray_directions, idx)}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "5BmrmyOXV7kD"
      },
      "outputs": [],
      "source": [
        "# Note: this cell takes a long time approx: 3 mins\n",
        "batch_size = 5000\n",
        "image_resolution = 800*800\n",
        "num_devices = jax.local_device_count()\n",
        "mapped_images_ds = batched_images_ds.map(lambda x: sample_data(x, image_size=image_resolution*num_images), num_parallel_calls=tf.data.AUTOTUNE)\n",
        "unbatched_mapped_ds = mapped_images_ds.unbatch()\n",
        "batched_rays_ds = unbatched_mapped_ds.batch(batch_size, num_parallel_calls=tf.data.AUTOTUNE).cache().prefetch(tf.data.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "9OGfheiBZsDB"
      },
      "outputs": [],
      "source": [
        "first_ray = tfds.as_numpy(unbatched_mapped_ds.take(1).get_single_element())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C7WE8GLsrkAb"
      },
      "source": [
        "## Train the Neural Radiance Field"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "H0qoFutV8X3y"
      },
      "outputs": [],
      "source": [
        "@partial(jax.jit, static_argnames=['randomized'])\n",
        "def mse(variables, ray_batch, rng, randomized=False):\n",
        "  # Define the squared loss for a single ray\n",
        "  def squared_error(ray, key):\n",
        "    actual = ray['color_image'].astype(float)\n",
        "    # TODO: clearly wrong to create keys here\n",
        "    rendered = dvgo.apply(\n",
        "        variables, rng=key, example=ray, randomized=randomized)\n",
        "    error = 256*rendered - actual\n",
        "    return jnp.abs(error)\n",
        "\n",
        "  # Vectorize to compute the average MSE loss for all rays.\n",
        "  num_rays = ray_batch['color_image'].shape[0]\n",
        "  key_batch = random.split(rng, num_rays)\n",
        "  return jnp.mean(jax.vmap(squared_error)(ray_batch, key_batch))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "ZBZ9hgkkha5G"
      },
      "outputs": [],
      "source": [
        "# Batched training\n",
        "rng = random.PRNGKey(42)\n",
        "key0, key1, rng = random.split(rng, 3)\n",
        "# We currently have to give init *one* ray, not a batch\n",
        "variables = dvgo.init(key0, rng=key1, example=first_ray)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "unpIFXVOaWs4",
        "outputId": "4cc7c8d5-56a9-42fd-a9ed-8199c498cc69"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "  0%|          | 10/12800 [00:30<7:45:48,  2.19s/it] "
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss step 0:  98.14512\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "  8%|▊         | 1009/12800 [00:42<02:33, 76.82it/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss step 1000:  11.8499365\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            " 16%|█▌        | 2000/12800 [00:55<04:59, 36.03it/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loss step 2000:  9.551577\n",
            "Completed training in 55.52354431152344 seconds.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "tx = optax.adam(learning_rate=0.03)\n",
        "opt_state = tx.init(variables)\n",
        "\n",
        "num_epochs = 1\n",
        "checkpoint_freq = 1000\n",
        "max_iterations = 2000\n",
        "loss_grad_fn = jax.value_and_grad(mse)\n",
        "\n",
        "tic = time.time()\n",
        "iter_count = 0\n",
        "for i in range(num_epochs):\n",
        "  for ray_batch in tqdm.tqdm(batched_rays_ds):\n",
        "    key, rng = random.split(rng)\n",
        "    loss_val, grads = loss_grad_fn(variables, \n",
        "                                   ray_batch=tfds.as_numpy(ray_batch), \n",
        "                                   rng=key, randomized=True)\n",
        "    updates, opt_state = tx.update(grads, opt_state)\n",
        "    variables = optax.apply_updates(variables, updates)\n",
        "    if iter_count % checkpoint_freq == 0:\n",
        "      print('Loss step {}: '.format(iter_count), loss_val)\n",
        "    iter_count += 1\n",
        "    if iter_count > max_iterations:  # Stopping at max_iterations on purpose to get an early trained model\n",
        "      break\n",
        "print(f'Completed training in {time.time() - tic} seconds.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqiVfW2yFsZn"
      },
      "source": [
        "## Examining the Result\n",
        "\n",
        "Using the now trained `variables`, we can render the views and compare them side-by-side to the ground truth:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 688
        },
        "id": "cx2AmS5e6n9f",
        "outputId": "0e7c7d29-47c6-4f77-be14-387ec7cf776c"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>GT</div><div><img width=\"200\" height=\"200\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>rendered</div><div><img width=\"200\" height=\"200\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>GT</div><div><img width=\"200\" height=\"200\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>rendered</div><div><img width=\"200\" height=\"200\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table class=\"show_images\" style=\"border-spacing:0px;\"><tr><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>GT</div><div><img width=\"200\" height=\"200\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td><td style=\"padding:1px;\"><div style=\"display:flex; align-items:left;\">\n",
              "      <div style=\"display:flex; flex-direction:column; align-items:center;\">\n",
              "      <div>rendered</div><div><img width=\"200\" height=\"200\" style=\"image-rendering:pixelated; object-fit:cover;\" src=\"\"/></div></div></div></td></tr></table>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "stride = 2\n",
        "for frame in ds.take(3):\n",
        "  image = frame['color_image']\n",
        "  rendered = render_image(variables, tfds.as_numpy(frame), stride)\n",
        "  media.show_images([image, rendered], \n",
        "                    titles=[\"GT\", \"rendered\"], \n",
        "                    height=400//stride)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "C9Mt-eC7zx8o"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "gB5bj5jP4oOu",
        "TlxfMEtHQCWF",
        "sl-2a8wIAH3j",
        "vUpDh2EEsEDh",
        "z60ukTmakvxr"
      ],
      "provenance": [],
      "machine_shape": "hm"
    },
    "gpuClass": "premium",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}